Virtual laser pointer as a point of contact indicator for machine learning assisted aerial refueling or other targeting

ABSTRACT

An example system includes a processor and a non-transitory computer-readable medium having stored therein instructions that are executable to cause the system to perform various functions. The functions include: (i) acquiring an image of a first aerial vehicle, the image depicting an object of a second aerial vehicle prior to contact between the object and a surface of the first aerial vehicle; (ii) providing the image as input to a data-driven analyzer that is trained in a supervised setting with example images for determining a predetermined point of contact between the object and the surface of the first aerial vehicle; (iii) determining, based on an output of the data-driven analyzer corresponding to the input, an estimated point of contact between the object and the surface of the first aerial vehicle; and (iv) providing the estimated point of contact to a display system.

FIELD

The present disclosure relates generally to aerial refueling, and moreparticularly, to systems and methods for using a virtual laser pointeras a point of contact indicator during aerial refueling.

BACKGROUND

One form of aerial refueling involves a complex targeting operationcombined with a controlled docking of a refueling boom from a supplyaircraft to a receptacle on a receiver aircraft. As part of thisoperation, an operator of the supply aircraft can use images from acamera to direct the refueling boom to dock with the receptacle on thereceiver aircraft. The operator can control an angle of the refuelingboom as well as a deployed length of the refueling boom.

SUMMARY

In one example, a system including a processor and a non-transitorycomputer-readable medium is described. The non-transitorycomputer-readable medium has stored therein instructions that areexecutable to cause the system to perform various functions. Thefunctions include acquiring an image of a first aerial vehicle. Theimage is acquired by a second aerial vehicle and depicts an object ofthe second aerial vehicle prior to contact between the object of thesecond aerial vehicle and a surface of the first aerial vehicle. Thefunctions also include providing the image as input to a data-drivenanalyzer that is trained in a supervised setting with example images fordetermining a predetermined point of contact between the object of thesecond aerial vehicle and the surface of the first aerial vehicle. Thedata-driven analyzer is configured to estimate an actual point ofcontact between the object of the second aerial vehicle and the surfaceof the first aerial vehicle based on features extracted from the image.In addition, the functions include determining, based on an output ofthe data-driven analyzer corresponding to the input, an estimated pointof contact between the object of the second aerial vehicle and thesurface of the first aerial vehicle. The functions further includeproviding the estimated point of contact to a display system.

In another example, a method is described. The method includes acquiringan image of a first aerial vehicle. The image is acquired by a secondaerial vehicle and depicts an object of the second aerial vehicle priorto contact between the object of the second aerial vehicle and a surfaceof the first aerial vehicle. The method also includes providing theimage as input to a data-driven analyzer that is trained in a supervisedsetting with example images for determining a predicted point of contactbetween the object of the second aerial vehicle and the surface of thefirst aerial vehicle. The data-driven analyzer is configured to estimatean actual point of contact between the object of the second aerialvehicle and the surface of the first aerial vehicle based on featuresextracted from the image. In addition, the method includes determining,based on an output of the data-driven analyzer corresponding to theinput, an estimated point of contact between the object of the secondaerial vehicle and the surface of the first aerial vehicle. Further, themethod includes overlaying an indication of the estimated point ofcontact on the surface of the first aerial vehicle within a video streamof the first aerial vehicle.

In another example, a non-transitory computer-readable medium isdescribed. The non-transitory computer-readable medium has storedtherein instructions that are executable to cause a system to performvarious functions. The functions include acquiring an image of a firstaerial vehicle. The image is acquired by a second aerial vehicle anddepicts an object of the second aerial vehicle prior to contact betweenthe object of the second aerial vehicle and a surface of the firstaerial vehicle. The functions also include providing the image as inputto a data-driven analyzer that is trained in a supervised setting withexample images for determining a predicted point of contact between theobject of the second aerial vehicle and the surface of the first aerialvehicle. The data-driven analyzer is configured to estimate an actualpoint of contact between the object of the first aerial vehicle and thesurface of the first aerial vehicle based on features extracted from theimage.

The features, functions, and advantages that have been discussed can beachieved independently in various examples or may be combined in yetother examples further details of which can be seen with reference tothe following description and figures.

BRIEF DESCRIPTION OF THE FIGURES

The novel features believed characteristic of the illustrative examplesare set forth in the appended claims. The illustrative examples,however, as well as a preferred mode of use, further objectives anddescriptions thereof, will best be understood by reference to thefollowing detailed description of an illustrative example of the presentdisclosure when read in conjunction with the accompanying figures,wherein:

FIG. 1 illustrates a first aerial vehicle and a second aerial vehicle,according to an example.

FIG. 2 illustrates a system, according to an example.

FIG. 3 illustrates a display of a video stream, according to an example.

FIG. 4 illustrates another display of a video stream, according to anexample.

FIG. 5 is a conceptual illustration of a training setup, according to anexample.

FIG. 6 shows a flowchart of a method, according to an example.

FIG. 7 shows an additional operation for use with the method shown inFIG. 6.

FIG. 8 shows additional operations for use with the method shown in FIG.6.

FIG. 9 shows additional operations for use with the method shown in FIG.6.

DETAILED DESCRIPTION

Disclosed examples will now be described more fully hereinafter withreference to the accompanying figures, in which some, but not all of thedisclosed examples are shown. Indeed, several different examples may beprovided and should not be construed as limited to the examples setforth herein. Rather, these examples are provided so that thisdisclosure will be thorough and complete and will fully convey the scopeof the disclosure to those skilled in the art.

As noted above, an operator of a supply aircraft can use images from acamera to direct a refueling boom to dock with a receptacle on areceiver aircraft. For instance, a display system of the supply aircraftcan display the images within a video stream to help the operatorcontrol the refueling boom. In practice, this task can be challengingdue to relevant movement between the supply aircraft and the receivercraft, poor lighting conditions, and the difficulty of trying tointerpret three-dimensional information from a camera that providestwo-dimensional information. For instance, it can be difficult for anoperator to perceive a depth between the refueling boom and thereceptacle, predict how much to deploy the refueling boom, and/or topredict how turbulence will affect movement of the refueling boom.Despite these difficulties, it is desirable that the refueling boomcontact only the receptacle, as impacting the receiving aircraft withthe refueling boom can damage both the receiver aircraft and therefueling boom.

Recent advances in machine learning have pushed the computer visionboundary. Specifically, through the utilization of supervised learningand large annotated datasets, machine learning models are becomingincreasingly more powerful, practical, and feasible for operationalintegration. An innovation that would greatly aid an operator in aerialrefueling is to know exactly where a refueling boom would contact areceiver aircraft if the refueling boom were extended at any given time.

Within examples, systems and methods for using machine learning topredict an estimated point of contact between a surface of a firstaerial vehicle and an object of a second aerial vehicle are described.In accordance with the present disclosure, after a computing systempredicts the estimated point of contact using a data-driven analyzer, adisplay system can then display an indication of the estimated point ofcontact within a video stream of the first aerial vehicle. For example,the display system can overlay an indication of the estimated point ofcontact on the surface of the first aerial vehicle within a video streamof the first aerial vehicle.

In an example method, an image of a first aerial vehicle can be acquiredby a second aerial vehicle. The image can depict an object of the secondaerial vehicle prior to contact between the object of the second aerialvehicle and a surface of the first aerial vehicle. For instance, thefirst aerial vehicle can be a receiver aircraft, the second aerialvehicle can be a supply aircraft, and the object can be a refuelingboom.

The image can then be provided to a data-driven analyzer that is trainedin a supervised setting. For instance, the data-driven analyzer can betrained with example images for determining a predicted point of contactbetween the object of the second aerial vehicle and a surface of thefirst aerial vehicle. The example images that the data-driven analyzeris trained on can include synthetic (e.g., simulated) images, laboratorygenerated images, or real images that are acquired during flight.Further, the data-driven analyzer can be configured to estimate anactual point of contact between the object of the second aerial vehicleand the surface of the first aerial vehicle based on features extractedfrom the image.

Based on an output of the data-driven analyzer corresponding to theinput, a computing system can determine an estimated point of contactbetween the object of the second aerial vehicle and the surface of thefirst aerial vehicle. A display system can then overlay an indication ofthe estimated point of contact on the surface of the first aerialvehicle within a video stream of the first aerial vehicle.

One technique of displaying the indication is to project a virtual laserpointer onto the first aerial vehicle at a location where the objectwould contact the surface of the first aerial vehicle if deployed. Abenefit of the virtual laser pointer approach is that a physical laserpointer can be incorporated into a scaled hardware mockup,computer-generated imagery, or actual flight imagery to aid in trainingof the data-driven analyzer, since the true position of the first aerialvehicle, the second aerial vehicle, and the object of the second aerialvehicle as well as the two-dimensional imagery that the operator sees,is available in these scenarios.

In one approach for generating training images, a laser pointer can beadded to a computer simulation that simulates relative motion betweenthe first aerial vehicle and the second aerial vehicle such thatcomputer-generated imagery of the surface of the first aerial vehicleincludes a dot or other indication produced by the laser. In a similarmanner, in a laboratory setting, a physical laser pointer can beincorporated into a scaled hardware mockup by attaching the physicallaser pointer to the object of the second aerial vehicle. Images of thesurface of the first aerial vehicle captured in the laboratory will theninclude a dot or other indication produced by the laser pointer when theprojection from the physical laser pointer intersects the surface of thefirst aerial vehicle. Additionally or alternatively, a physical laserpointer can be added to the object of the second aerial vehicle for thepurpose of generating training images of the surface of the first aerialvehicle during flight.

Another benefit of the virtual laser pointer approach to representingthe estimated point of contact is that this feature can be added to adisplay system of the second aerial vehicle without having to modify thesecond aerial vehicle to include additional hardware. During flight, thedata-driven analyzer can estimate the point of contact and provide theestimate to a display system, such that the display system can overlayan indication of the point of contact on the video stream the operatoris viewing. In this manner, the operator can more easily predict andcontrol deployment of the object.

In some instances, the data-driven analyzer can be configured toestimate additional information as well, such as a separation distancebetween the object of the second aerial vehicle and the surface of thefirst aerial vehicle and/or a confidence of the estimated point ofcontact between the object and the surface of the first aerial vehicle.Indications of the separation distance and/or confidence can also bedisplayed within the video stream for use by an operator in controllingdeployment of the object.

Various other features of these systems and methods are describedhereinafter with reference to the accompanying figures.

Referring now to FIG. 1, FIG. 1 illustrates a first aerial vehicle 102and a second aerial vehicle 104, utilizing a system according to exampleimplementations of the present disclosure. As shown, first aerialvehicle 102 can take the form of a receiver aircraft, and second aerialvehicle 104 can take the form of a supply aircraft that is refueling thereceiver aircraft using a refueling boom. Second aerial vehicle 104 caninclude a system for aiding maneuvering of the refueling boom.

FIG. 2 illustrates a system 200 for aiding maneuvering of an object ofan aerial vehicle, according to example implementation of the presentdisclosure. As shown in FIG. 2, system 200 includes a camera 202, acontroller 204, and a display system 206. In line with the discussionabove, system 200 can be positioned within a supply aircraft having arefueling boom. Camera 202, controller 204, and display system 206 maybe co-located or directly coupled to one another, or in some examples,they may communicate with one another across one or more computernetworks. For instance, camera 202, controller 204, and display system206 can be in wired or wireless communication with each other by way ofone or more communication links or in wired or in wired or wirelesscommunication with a central computing device.

In some examples, camera 202 is configured to obtain one or more imagesof a first aerial vehicle, such as a receiver aircraft having areceptacle for refueling the receiver aircraft. The receiver aircraftmay be in flight and the supply aircraft having the system 200 may alsobe in flight to perform the task of refueling the receiver aircraftthrough a refueling boom. Camera 202 may acquire the images, andcontroller 204 may obtain the images from camera 202. Camera 202 caninclude a charge-coupled device (CCD) camera, for instance.

Controller 204 can take the form of a control unit, laptop computer,mobile computer, wearable computer, tablet computer, desktop computer,or other type of computing device. As such, controller 204 includes aprocessor 208 and a memory 210. Processor 208 could be any type ofprocessor, such as a microprocessor, digital signal processor, multicoreprocessor, etc. Memory 210 can include a non-transitory computerreadable medium (CRM) 214 storing program instructions that areexecutable by processor 208 or a group of processors to carry out any ofthe controller functions described herein. Controller 204 can furtherinclude an input device and one or more communication ports throughwhich controller 204 is configured to communicate with other componentsof system 200 or other devices that are external to system 200.

After obtaining one or more images from camera 202, controller 204 isconfigured to estimate an actual point of contact between the object ofthe second aerial vehicle and a surface of the first aerial vehicleusing the one or more images. In particular, controller 204 can providethe one or more images as input to a data-driven analyzer 212 stored bymemory 210. Data-driven analyzer 212 is trained in a supervised settingwith example images for determining a predicted point of contact betweenthe object of the second aerial vehicle and a surface of the firstaerial vehicle.

More particularly, data-driven analyzer 212 can include a model that istrained based on supervised (or semi-supervised) learning via maximumlikelihood estimation. This learning can include providing exampleimages to the model such that the model can learn with maximumlikelihood where a point of contact will be.

As one example, the data-driven analyzer can include a parameterizedmodel that is trained in a supervised setting by introducing a set ofexample images including an object of a second aerial vehicle and asurface of the a first aerial vehicle that provides an output of apredicted point of contact between the object of the second aerialvehicle and the surface of the first aerial vehicle. With this approach,the parametrized model is configured to estimate an actual point ofcontact between the object of the second aerial vehicle and the surfaceof the first aerial vehicle based on features extracted from an imageacquired by camera 202.

As another example, the data-driven analyzer can include a neuralnetwork that is trained in a supervised setting by introducing a set ofexample images including an object of a second aerial vehicle and asurface of the a first aerial vehicle that provides an output of apredicted point of contact between the object of the second aerialvehicle and the surface of the first aerial vehicle. One example of aneural network is a convolutional neural network. With this approach,the neural network is configured to estimate an actual point of contactbetween the object of the second aerial vehicle and the surface of thefirst aerial vehicle based on features extracted from an image acquiredby camera 202.

Further, data-driven analyzer 212 can be configured to utilize either asingle image or a sequence of images as input, depending on the desiredimplementation. In the single-image approach, the trained model can takeas input an image of a first aerial vehicle and an object of a secondaerial vehicle, and predict, within a two-dimensional image, a point ofcontact between the surface of the first aerial vehicle and the objectof the second aerial vehicle. Similarly, in the multiple-image approach(i.e. time-dependent), the trained model can take as input a sequence ofimages of a first aerial vehicle and an object of a second aerialvehicle, and predict, within a two-dimensional image, a point of contactbetween the surface of the first aerial vehicle and the object of thesecond aerial vehicle.

In line with the discussion above, the example images that data-drivenanalyzer 212 is trained on can be generated by incorporating a physicallaser pointer into a scaled hardware mockup, computer-generated imagery,or actual flight imagery to aid in training of data-driven analyzer 212.As one example, a laser pointer can be added to a computer simulationthat simulates relative motion between the first aerial vehicle and thesecond aerial vehicle such that computer-generated imagery of thesurface of the first aerial vehicle includes a dot or other indicationproduced by the laser. The laser pointer can be coupled to an end of theobject and oriented such that the laser outputs a beam in a directionthat is parallel to a longitudinal axis of the object. As anotherexample, in a laboratory setting, a physical laser pointer can beincorporated into a scaled hardware mockup by attaching the physicallaser pointer to the object of the second aerial vehicle. Images of thesurface of the first aerial vehicle acquired in the laboratory will theninclude a dot or other indication produced by the laser pointer at theintersection between the beam and the surface of the first aerialvehicle. As a further example, a physical laser pointer can be added tothe object of the second aerial vehicle for the purpose of generatingtraining images of the surface of the first aerial vehicle duringflight.

Various techniques can be used to train data-driven analyzer 212. By wayof example, data-driven analyzer 212 can be trained in a supervisedsetting with stochastic gradient descent using either a least-squares L₂loss cross-entropy (log-loss), or a more sophisticated generativeadversarial network (GAN) loss. For this training, software tools suchas Tensorflow, PyTorch, etc., can be used.

Data-driven analyzer 212 is configured to estimate an actual point ofcontact between the object and the surface of the first aerial vehiclebased on features extracted from the one or more images. The featuresextracted from the one or more images can be either actual outputfeatures, i.e., keypoints, or can be hidden/latent parameters of aneural network. In either case, the extracted features can be learnedthrough minimizing of the loss during training.

Data-driven analyzer 212 can also be configured to estimate the actualpoint of contact based on an orientation and position of the firstaerial vehicle relative to the second aerial vehicle. With thisapproach, controller 204 can derive the orientation and position fromone or more images acquired by camera 202, and provide the orientationand position as part of the input to data-driven analyzer 212.Determining the orientation and position can involve determining, frompoints on an image of the first aerial vehicle, an orientation andposition of the first aerial vehicle relative to camera 202 using a poseestimation algorithm and a known relationship between the points on theimage and corresponding points on a three-dimensional model of the firstaerial vehicle. The three-dimensional model of the first aerial vehiclecan be stored in memory 210.

In some examples, data-driven analyzer 212 is also be configured tooutput a confidence of the estimated point of contact. Additionally oralternatively, data-driven analyzer 212 can be configured to estimate aseparation distance between the object of the second aerial vehicle andthe surface of the first aerial vehicle (e.g., a separation distancebetween a tip of the refueling boom and a surface of the receiveraircraft).

Controller 204 can provide the estimated point of contact to displaysystem 206. Display system 206 is configured to display a video streamof the first aerial vehicle.

In one example, data-driven analyzer 212 can output a two-dimensionalimage that includes an indication of the estimated point of contact, andcontroller 204 can analyze the image to determine coordinates (e.g., anx-y position) of the estimated point of contact. Alternatively,data-driven analyzer 212 can directly output coordinates of theestimated point of contact. Controller 204 can provide the estimatedpoint of contact to a rendering module of display system 206, and therendering module can then overlay an indication of the estimated pointof contact on a video stream of the first aerial vehicle. In examples inwhich data-driven analyzer 212 outputs a confidence in the estimatedpoint of contact and/or an estimated separation distance, controller 204can also provide the confidence and/or estimated separation distance todisplay system 206. The rendering module can then overlay indications ofthe confidence and/or estimated separation distance on the video streamof the first aerial vehicle. The indication of the confidence caninclude a size of a shape surrounding the indication of the estimatedpoint of contact. Additionally or alternatively, the indication of theconfidence can include a color of the indication of the estimated pointof contact.

FIG. 3 illustrates a display of a video stream 300, according to anexample. As shown in FIG. 3, an indication 302 of an estimated point ofcontact between a refueling boom 304 and a surface of a receiveraircraft 306 is overlaid on video stream 300. In FIG. 3, indication 302is shown as a virtual laser beam output from a tip 308 of refueling boom304 and in a direction that is parallel to a longitudinal axis ofrefueling boom 304. In this example, the operator of a supply aircraftis attempting to guid tip 308 of refueling boom 304 into a receptacle310 on receiver aircraft 306. Indication 302 intersects with receptacle310. The operator can interpret this to mean that, upon furtherdeployment of refueling boom 304, tip 308 is likely to be receivedwithin receptacle 310.

In some examples, indication 302 can include an indication of aconfidence of the estimated point of contact. For example, a color ofindication 302 can be varied such that the color is indicative of aconfidence of the estimated point of contact. With this approach, anoperator can interpret the fact that indication 302 is red to mean thatthe confidence is low (or below a threshold) and can interpret the factthat indication 302 is green to meant that the confidence is high (orabove a threshold).

As further shown in FIG. 3, an indication 312 of an estimated separationdistance (ESD) between tip 308 and the surface of receiver aircraft 306is also displayed within video stream. In this example, the ESD isfifteen feet. Indication 312 can aid the operator in determining howmuch further to deploy refueling boom 304.

FIG. 4 illustrates another display of a video stream 400, according toan example. Like video stream 300 of FIG. 3, video stream 400 includesan indication 402 of an estimated point of contact between a refuelingboom 404 and a surface of a receiver aircraft 406. However, unlikeindication 302 of FIG. 3, indication 402 includes a shape that surroundsindication 402 and serves as an indication of a confidence of theestimated point of contact. In particular, FIG. 4 shows indication 402as including a virtual cone surrounding a virtual laser beam output froma tip 408 of refueling boom 404. A size of the shape can be varied suchthat the size is indicative of the confidence of the estimated point ofcontact. For instance, an operator can interpret the fact thatindication 402 includes a large virtual cone to mean that the confidenceis low and can interpret the fact that indication 402 includes a smallvirtual cone to mean that the confidence is high.

FIG. 5 is a conceptual illustration 500 of a training setup, accordingto an example. In line with the discussion above, training images fortraining a data-driven analyzer can be acquired in a laboratory using ahardware mockup that includes a scaled model 502 of a refueling boom, ascaled model 504 of a receiver aircraft, a laser pointer 506, and acamera 508. Laser pointer 506 is coupled to model 502 such that a laserbeam output by laser pointer 506 is aligned with a longitudinal axis ofthe refueling boom and intersects a surface of model 504 at a locationat which the refueling boom would impact the surface if the refuelingboom were further deployed. Further, camera 508 is provided inappropriate position and orientation that corresponds to the positionand orientation of a camera on a supply aircraft relative to therefueling boom of the supply aircraft.

In operation, a laser beam output by laser pointer 506 intersects with asurface of model 504, producing a dot at the intersection between thelaser beam and a surface of model 504. Camera 508 then acquire images asactuators 510 are manipulated (e.g., either manually orprogrammatically) to vary a position of model 504 of the receiveraircraft relative to model 502 of the refueling boom. In this manner, aplurality of training images can be rapidly generated, for use intraining a data-driven analyzer.

In some examples, a separation distance between the refueling boom andthe surface of the receiver aircraft can be measured and stored inassociation with respective training images. This information can beuseful for training the data-driven analyzer to estimate a separationdistance between the refueling boom and the surface of the receiveraircraft.

FIG. 6 shows a flowchart of a method 600, according to an example.Method 600 shown in FIG. 6 presents an embodiment of a method that, forexample, could be used with system 200 of FIG. 2, for example, or any ofthe systems disclosed herein. Any of the example devices or systemsdescribed herein, such as components of system 200 of FIG. 2, may beused or configured to perform logical functions presented in FIG. 6.

Method 600 can include one or more operations, functions, or actions asillustrated by one or more of blocks 602-608. Although these blocks areillustrated in a sequential order, these blocks may also be performed inparallel, and/or in a different order than those described herein. Also,the various blocks may be combined into fewer blocks, divided intoadditional blocks, and/or removed based upon the desired implementation.

It should be understood that for this and other processes and methodsdisclosed herein, flowcharts show functionality and operation of onepossible implementation of present embodiments. In this regard, eachblock may represent a module, a segment, or a portion of program code,which includes one or more instructions executable by a processor forimplementing specific logical functions or steps in the process. Theprogram code may be stored on any type of computer readable medium ordata storage, for example, such as a storage device including a disk orhard drive. The computer readable medium may include non-transitorycomputer readable medium or memory, for example, such as computerreadable media that stores data for short periods of time like registermemory, processor cache, and RAM. The computer readable media may alsobe any other volatile or non-volatile storage systems. The computerreadable medium may be considered a tangible computer readable storagemedium, for example.

Initially, at block 602, method 600 includes acquiring an image of afirst aerial vehicle. The image is acquired by a second aerial vehicleand depicts an object of the second aerial vehicle prior to contactbetween the object of the second aerial vehicle and a surface of thefirst aerial vehicle. For example, the image can depict a receiveraircraft and a refueling boom of a supply aircraft, prior to contactbetween the refueling boom and a surface of the receiver aircraft. Theimage can be acquired during a refueling operation and while thereceiver aircraft and supply aircraft are in flight.

At block 604, method 600 includes providing the image as input to adata-driven analyzer that is trained in a supervised setting withexample images for determining a predicted point of contact between theobject of the second aerial vehicle and the surface of the first aerialvehicle. The data-driven analyzer is configured to estimate an actualpoint of contact between the object of the second aerial vehicle and thesurface of the first aerial vehicle based on features extracted from theimage.

At block 606, method 600 includes determining, based on an output of thedata-driven analyzer corresponding to the input, an estimated point ofcontact between the object of the second aerial vehicle and the surfaceof the first aerial vehicle.

And at block 608, method 600 includes overlaying an indication of theestimated point of contact on the surface of the first aerial vehiclewithin a video stream of the first aerial vehicle.

FIG. 7 shows an additional operation for use with the method shown inFIG. 6. Block 702 of FIG. 7 could be performed as part of block 608 ofFIG. 6. At block 702, FIG. 7 includes displaying an indication of aconfidence of the estimated point of contact within the video stream.For instance, the indication of the confidence can include a size of ashape surrounding the indication of the estimated point of contactand/or a color of the indication of the estimated point of contact.

FIG. 8 shows additional operations for use with the method shown in FIG.6. Blocks 802 and 804 of FIG. 8 could be performed as part of block 602of FIG. 6. At block 802, FIG. 8 includes acquiring a sequence of imagesthat depict the first aerial vehicle and the object of the second aerialvehicle prior to contact between the object of the second aerial vehicleand the surface of the first aerial vehicle. Further, at block 804, FIG.8 includes providing the sequence of images as the input to thedata-driven analyzer. Using a sequence of images rather than a singleimage can improve the estimate of the point of contact, since thesequence of images can inherently include information about relativemotion between the first aerial vehicle and the object of the secondaerial vehicle.

FIG. 9 shows additional operations for use with the method shown in FIG.6. Blocks 902 and 904 of FIG. 9 could be performed before, after, or inparallel with block 606 of FIG. 6, for instance. At block 904, FIG. 9includes determining, based on the output of the data-driven analyzer,an estimated separation distance between the object of the second aerialvehicle and the surface of the first aerial vehicle.

The description of the different advantageous arrangements has beenpresented for purposes of illustration and description, and is notintended to be exhaustive or limited to the examples in the formdisclosed. After reviewing and understanding the foregoing disclosure,many modifications and variations will be apparent to those of ordinaryskill in the art. Further, different examples may provide differentadvantages as compared to other examples. The example or examplesselected are chosen and described in order to best explain theprinciples, the practical application, and to enable others of ordinaryskill in the art to understand the disclosure for various examples withvarious modifications as are suited to the particular use contemplated.

What is claimed is:
 1. A system comprising: a processor; and anon-transitory computer-readable medium having stored thereininstructions that are executable to cause the system to performfunctions comprising: acquiring an image of a first aerial vehicle,wherein the image is acquired by a second aerial vehicle and depicts anobject of the second aerial vehicle prior to contact between the objectof the second aerial vehicle and a surface of the first aerial vehicle,providing the image as input to a data-driven analyzer that is trainedin a supervised setting with example images for determining a predictedpoint of contact between the object of the second aerial vehicle and thesurface of the first aerial vehicle, wherein the data-driven analyzer isconfigured to estimate an actual point of contact between the object ofthe second aerial vehicle and the surface of the first aerial vehiclebased on features extracted from the image; determining, based on anoutput of the data-driven analyzer corresponding to the input, anestimated point of contact between the object of the second aerialvehicle and the surface of the first aerial vehicle; and providing theestimated point of contact to a display system.
 2. The system of claim1, wherein the data-driven analyzer comprises a parameterized model thatis trained in a supervised setting by introducing a set of exampleimages including the object of the second aerial vehicle and the surfaceof the first aerial vehicle that provides an output of a predicted pointof contact between the object of the second aerial vehicle and thesurface of the first aerial vehicle, the parameterized model beingconfigured to estimate an actual point of contact between the object ofthe second aerial vehicle and the surface of the first aerial vehiclebased on features extracted from the image.
 3. The system of claim 1,wherein the data-driven analyzer comprises a neural network that istrained in a supervised setting by introducing a set of example imagesincluding the object of the second aerial vehicle and the surface of thefirst aerial vehicle that provides an output of a predicted point ofcontact between the object of the second aerial vehicle and the surfaceof the first aerial vehicle, the neural network being configured toestimate an actual point of contact between the object of the secondaerial vehicle and the surface of the first aerial vehicle based onfeatures extracted from the image.
 4. The system of claim 1, wherein:the first aerial vehicle is a receiver aircraft, the second aerialvehicle is a supply aircraft, and the object of the second aerialvehicle is a refueling boom.
 5. The system of claim 1, furthercomprising a camera configured to acquire the image of the second aerialvehicle.
 6. The system of claim 1, further comprising the displaysystem, wherein the display system is configured to display a videostream of the first aerial vehicle and to overlay an indication of theestimated point of contact on the surface of the first aerial vehiclewithin the video stream.
 7. The system of claim 6, wherein: the outputof the data-driven analyzer corresponding to the input comprises aconfidence of the estimated point of contact, and the indication of theestimated point of contact comprises an indication of the confidence. 8.The system of claim 7, wherein the indication of the confidencecomprises a size of a shape surrounding the indication of the estimatedpoint of contact.
 9. The system of claim 7, wherein the indication ofthe confidence comprises a color of the indication of the estimatedpoint of contact.
 10. The system of claim 1: wherein the functionsfurther comprise: acquiring a sequence of images that depict the firstaerial vehicle and the object of the second aerial vehicle prior tocontact between the object of the second aerial vehicle and the surfaceof the first aerial vehicle; and providing the sequence of images as theinput to the data-driven analyzer, wherein the image is part of thesequence of images, and wherein the data-driven analyzer is configuredto estimate an actual point of contact based on features extracted fromthe sequence of images.
 11. The system of claim 1: wherein the functionsfurther comprise: acquiring an orientation and position of the firstaerial vehicle relative to the second aerial vehicle; and providing theorientation and position as part of the input to the data-drivenanalyzer, and wherein the data-driven analyzer is configured to estimatean actual point of contact based on the orientation and position. 12.The system of claim 11, wherein acquiring the orientation and positioncomprises deriving the orientation and position from the image.
 13. Thesystem of claim 1: wherein the data-driven analyzer is configured toestimate a separation distance between the object of the second aerialvehicle and the surface of the first aerial vehicle, and wherein thefunctions further comprise: determining, based on the output of thedata-driven analyzer, an estimated separation distance between theobject of the second aerial vehicle and the surface of the first aerialvehicle; and providing the estimated separation distance to the displaysystem.
 14. A method comprising: acquiring an image of a first aerialvehicle, wherein the image is acquired by a second aerial vehicle anddepicts an object of the second aerial vehicle prior to contact betweenthe object of the second aerial vehicle and a surface of the firstaerial vehicle; providing the image as input to a data-driven analyzerthat is trained in a supervised setting with example images fordetermining a predicted point of contact between the object of thesecond aerial vehicle and the surface of the first aerial vehicle,wherein the data-driven analyzer is configured to estimate an actualpoint of contact between the object of the second aerial vehicle and thesurface of the first aerial vehicle based on features extracted from theimage; determining, based on an output of the data-driven analyzercorresponding to the input, an estimated point of contact between theobject of the second aerial vehicle and the surface of the first aerialvehicle; and overlaying an indication of the estimated point of contacton the surface of the first aerial vehicle within a video stream of thefirst aerial vehicle.
 15. The method of claim 14, wherein: the firstaerial vehicle is a receiver aircraft, the second aerial vehicle is asupply aircraft, and the object of the second aerial vehicle is arefueling boom.
 16. The method of claim 14, wherein: the output of thedata-driven analyzer corresponding to the input comprises a confidenceof the estimated point of contact, and the indication of the estimatedpoint of contact comprises an indication of the confidence.
 17. Themethod of claim 14, further comprising: acquiring a sequence of imagesthat depict the first aerial vehicle and the object of the second aerialvehicle prior to contact between the object of the second aerial vehicleand the surface of the first aerial vehicle; and providing the sequenceof images as the input to the data-driven analyzer, wherein the image ispart of the sequence of images, and wherein the data-driven analyzer isconfigured to estimate an actual point of contact based on featuresextracted from the sequence of images.
 18. The method of claim 14:wherein the data-driven analyzer is configured to estimate a separationdistance between the object of the second aerial vehicle and the surfaceof the first aerial vehicle, and wherein the method further comprises:determining, based on the output of the data-driven analyzer, anestimated separation distance between the object of the second aerialvehicle and the surface of the first aerial vehicle; and displaying theestimated separation distance within the video stream.
 19. Anon-transitory computer-readable medium having stored thereininstructions that are executable to cause a system to perform functionscomprising: acquiring an image of a first aerial vehicle, wherein theimage is acquired by a second aerial vehicle and depicts an object ofthe second aerial vehicle prior to contact between the object of thesecond aerial vehicle and a surface of the first aerial vehicle;providing the image as input to a data-driven analyzer that is trainedin a supervised setting with example images for determining a predictedpoint of contact between the object of the second aerial vehicle and thesurface of the first aerial vehicle, wherein the data-driven analyzer isconfigured to estimate an actual point of contact between the object ofthe second aerial vehicle and the surface of the first aerial vehiclebased on features extracted from the image; determining, based on anoutput of the data-driven analyzer corresponding to the input, anestimated point of contact between the object of the second aerialvehicle and the surface of the first aerial vehicle; and overlaying anindication of the estimated point of contact on the surface of the firstaerial vehicle within a video stream of the first aerial vehicle. 20.The non-transitory computer-readable medium of claim 19, wherein: thefirst aerial vehicle is a receiver aircraft, the second aerial vehicleis a supply aircraft, and the object of the second aerial vehicle is arefueling boom.